User feedback on the NHS Test & Trace Service during COVID-19: the use of machine learning to analyse free-text data from 37,914 UK adults (preprint)

Objectives The UK government's approach to the pandemic relies on a test, trace and isolate strategy, mainly implemented via the digital NHS Test & Trace Service. Feedback on user experience is central to the successful development of public-facing services. As the situation dynamically changes and data accumulate, interpretation of feedback by humans becomes time-consuming and unreliable. The specific objectives were to 1) evaluate a human-in-the-loop machine learning technique based on structural topic modelling in terms of its serviceability in the analysis of vast volumes of free-text data, 2) generate actionable themes that can be used to increase user satisfaction of the Service. Methods We evaluated an unsupervised Topic Modelling approach, testing models with 5-40 topics and differing covariates. Two human coders conducted thematic analysis to interpret the topics. We identified a Structural Topic Model with 25 topics and metadata as covariates as the most appropriate for acquiring insights. Results Results from analysis of feedback by 37,914 users from May 2020 to March 2021 highlighted issues with the Service falling within three major themes: multiple contacts and incompatible contact method and incompatible contact method, confusion around isolation dates and tracing delays, complex and rigid system. Conclusions Structural Topic Modelling coupled with thematic analysis was found to be an effective technique to rapidly acquire user insights. Topic modelling can be a quick and cost-effective method to provide high quality, actionable insights from free-text feedback to optimize public health services.

Applying machine-learning to rapidly analyse large qualitative text datasets to inform the COVID-19 pandemic response: Comparing human and machine-assisted topic analysis techniques (preprint)

Background: Machine-assisted topic analysis (MATA) uses artificial intelligence methods to assist qualitative researchers to analyse large amounts of textual data. This could allow qualitative researchers to inform and update public health interventions 'in real-time', to ensure they remain acceptable and effective during rapidly changing contexts (such as a pandemic). Objective: We aimed to understand the potential for such approaches to support intervention implementation, by directly comparing MATA and 'human-only' thematic analysis techniques when applied to the same dataset (1472 free-text responses from users of the COVID-19 infection control intervention 'Germ Defence'). Methods: In MATA, the analysis process included an unsupervised topic modelling approach to identify latent topics in the text. The human research team then described the topics and identified broad themes. In human-only codebook analysis, an initial codebook was developed by an experienced qualitative researcher and applied to the dataset by a well-trained research team, who met regularly to critique and refine the codes. To understand similarities and difference, formal triangulation using a 'convergence coding matrix' compared the findings from both methods, categorising them as 'agreement', 'complementary', 'dissonant', or 'silent'. Results: Human analysis took much longer (147.5 hours) than MATA (40 hours). Both human-only and MATA identified key themes about what users found helpful and unhelpful (e.g. Helpful: Boosting confidence in how to perform the behaviours. Unhelpful: Lack of personally relevant content). Formal triangulation of the codes created showed high similarity between the findings. All codes developed from the MATA were classified as in agreement or complementary to the human themes. Where the findings were classified as complementary, this was typically due to slightly differing interpretations or nuance present in the human-only analysis. Conclusions: Overall, the quality of MATA was as high as the human-only thematic analysis, with substantial time savings. For simple analyses that do not require an in-depth or subtle understanding of the data, MATA is a useful tool that can support qualitative researchers to interpret and analyse large datasets quickly. These findings have practical implications for intervention development and implementation, such as enabling rapid optimisation during public health emergencies.

Location, Location, Location: A Discrete Choice Experiment to Inform Vaccination Programme Delivery in the UK (preprint)

Background: Large-scale vaccination is fundamental to combatting COVID-19. In March 2021, the UK’s vaccination programme had delivered vaccines to large proportions of older and more vulnerable population groups; however, there was concern that uptake would be lower among young people. This research was designed to elicit the preferences of 18-29-year-olds with respect to key delivery characteristics. Methods: From 25 March - 2 April 2021, an online sample of 2,021 UK adults aged 18-29 years participated in a Discrete Choice Experiment. Participants made six choices, each between two SMS invitations to get vaccinated; each choice also had an opt-out. Each invitation had four attributes (1 x 5 levels, 3 x 3 levels): delivery mode, appointment timing, proximity, and SMS sender. These were systematically varied according to a d-optimal fractional factorial design. Order of presentation was randomised for each participant. Responses were analysed using a mixed logit model. Results: The logit model revealed a large alternative-specific constant (β = 1.385, SE = 0.067, p <0.001), indicating a strong preference for ‘opting in’ to appointment invitations. Pharmacies were dispreferred to the local vaccination centre (β = -0.256, SE = 0.072, p <0.001), appointments in locations that were 30-45 minutes travel time from one’s premises were dispreferred to locations that were less than 15 minutes away (β = -0.408, SE = 0.054, p <0.001), and, compared to invitations sent by the NHS, SMSs forwarded by ‘a friend’ were dispreferred (β = -0.615, SE = 0.056, p <0.001) but invitations from the General Practitioner were preferred (β = 0.105, SE = 0.048, p = 0.028). Conclusions: The results indicated that the existing configuration of the UK’s mass vaccination programme was well-placed to deliver vaccines to 18-29-year-olds; however, some adjustments might enhance acceptance. Local pharmacies were not preferred; long travel times were a disincentive but close proximity (0-15 minutes from one’s premises) was not necessary; and either the ‘NHS’ or ‘Your GP’ would serve as adequate invitation sources. This research informed COVID-19 policy in the UK, and contributes to a wider body of Discrete Choice Experiment evidence on citizens’ preferences, requirements and predicted behaviours regarding COVID-19.

Perceived risk factors for severe Covid-19 symptoms and their association with health behaviours: Findings from the HEBECO study (preprint)

Background: There remains uncertainty about Covid-19 risk factors. We examined UK adults’ risk perceptions for severe Covid-19 symptoms and whether engaging in concurrent health behaviours is associated with risk perceptions. Methods: Cross-sectional analysis of data from the HEBECO study where 2206 UK adults classified potential factors (age 70+, ethnic minority, medical comorbidities, vaping, smoking cigarettes, alcohol drinking, regular physical activity, being overweight, eating unhealthy foods, using nicotine replacement therapy – NRT, lower income, poor housing, being a keyworker) as either increasing, decreasing, or having no impact on severe Covid-19 symptoms. Logistic regressions examined whether engaging in health behaviours was associated with risk perceptions after adjusting for socio-demographic characteristics, health conditions and other behaviours. Results: The great majority (89-99%) of adults classified age 70+, having comorbidities, being a key worker, overweight, and from an ethnic minority as increasing the risk. People were less sure about alcohol drinking, vaping, and nicotine replacement therapy use (17.4-29.5% responding ‘don’t know’). Relative to those who did not, those who smoked tobacco, vaped and consumed alcohol had significantly (all p<0.015) higher odds (aORs=1.58 to 5.80) for classifying these behaviours as ‘no impact’ or ‘decreasing risk’, and lower odds (aORs=.25 to .72) for classifying as ‘increasing risk’. Similarly, eating more fruit and vegetables was associated with classifying unhealthy diet as ‘increasing risk’ (aOR=1.37,1.12-1.69), and exercising more with classifying regular physical activity as ‘decreasing risk’ (aOR=2.42,1.75-3.34). Conclusions: Risk perceptions for severe Covid-19 symptoms were lower for adults’ own health behaviours, evidencing optimism bias.Implications: These risk perceptions may form barriers to changing one’s own unhealthy behaviours or make one less responsive to interventions that refer to the risk of Covid-19 as a motivating factor. Thus, in some cases risk perceptions could help sustain unhealthy behaviours and exacerbate inequalities in health behaviours and outcomes.

Effectiveness of digital interventions to improve household and community infection prevention and control behaviours and to reduce incidence of respiratory and/or gastro-intestinal infections: A rapid systematic review (preprint)

BackgroundDigital interventions have potential to efficiently support improved hygiene practices to reduce transmission of COVID-19. ObjectiveTo evaluate the evidence for digital interventions to improve hygiene practices within the community. MethodsWe reviewed articles published between 01 January 2000 and 26 May 2019 that presented a controlled trial of a digital intervention to improve hygiene behaviours in the community. We searched MEDLINE, Embase, PsycINFO, Cochrane Controlled Register of Trials (CENTRAL), China National Knowledge Infrastructure and grey literature. Trials in hospitals were excluded, as were trials aiming at prevention of sexually transmitted infections; only target diseases with transmission mechanisms similar to COVID-19 (e.g. respiratory and gastrointestinal infections) were included. Trials had to evaluate a uniquely digital component of an intervention. Study designs were limited to randomised controlled trials, controlled before-and-after trials, and interrupted time series analyses. Outcomes could be either incidence of infections or change in hygiene behaviours. The Risk of Bias 2 tool was used to assess study quality. ResultsWe found seven studies that met the inclusion criteria. Six studies reported successfully improving self-reported hygiene behaviour or health outcomes, but only one of these six trials confirmed improvements using objective measures (reduced consultations and antibiotic prescriptions), Germ Defence. Settings included kindergartens, workplaces, and service station restrooms. Modes of delivery were diverse: WeChat, website, text messages, audio messages to mobiles, electronic billboards, and electronic personal care records. Four interventions targeted parents of young children with educational materials. Two targeted the general population; these also used behaviour change techniques or theory to inform the intervention. Only one trial had low risk of bias, Germ Defence; the most common concerns were lack of information about the randomisation, possible bias in reporting of behavioural outcomes, and lack of an analysis plan and possible selective reporting of results. ConclusionThere was only one intervention that was judged to be at low risk of bias, Germ Defence, which reduced incidence and severity of illness, as confirmed by objective measures. Further evaluation is required to determine the effectiveness of the other interventions reviewed.

Current infection control behaviour patterns in the UK, and how they can be improved by 'Germ Defence', an online behavioural intervention to reduce the spread of COVID-19 in the home. (preprint)

Background: Germ Defence (https://germdefence.org/) is a freely available website providing behavioural advice for infection control within households, using behaviour change techniques. This observational study reports current infection control behaviours in the home in UK and international users of the website, and examine how they might be improved to reduce the spread of COVID-19. Method: 28,285 users sought advice from four website pathways (to protect themselves generally, to protect others if the user was showing symptoms, to protect themselves if household members were showing symptoms, and to protect a household member who is at high risk) and completed outcome measures of current infection control behaviours within the home (self-isolation, social distancing, putting shopping/packages aside, wearing face-covering, cleaning and disinfecting, handwashing), and intentions to change these behaviours. Results: Current user behaviours mean scores varied across all infection control measures but were between 'sometimes' and 'quite often', except handwashing ('very often'). Behaviours were similar regardless of the website pathway used. After using Germ Defence, users recorded intentions to improve infection control behaviour across all website pathways and for all behaviours. Conclusions: Self-reported infection control behaviours other than handwashing are lower than is optimal for infection prevention, although reported handwashing is much higher. The advice using behaviour change techniques in Germ Defence led to intentions to improve these behaviours. This has been shown previously to reduce the incidence, severity and transmission of infections. These findings suggest that promoting Germ Defence within national and local public health guidance could reduce COVID-19 transmission.